This time I was lazy and kept USERCMD_MSEC and USERCMD_HZ around, just turning them into #defines that alias global variables storing the current values.
Not sure how great that really is, but at least it reduces the amount of changed code significantly, as most (but unfortunately not all) code using USERCMD_MSEC/HZ just continues to work without modification.

Ok, I think I have fixed the "Switching game modes can trigger a fake black screen" issue, by fixing how com_frameTime is calculated.
I couldn't reproduce the issue with the grabber either (after the fix, didn't try before).

At least on my (Linux) PC, the framerate is extremely stable now (with vsync disabled), but not at the rate configured with com_gameHz, but the next multiple of int(1000/com_gameHz), e.g. 63fps for 60hz, 125fps for 120hz, 143fps for 144Hz, 250fps for 240Hz.

I guess I should try using floating point (double) times at least for the async tics, but probably I should also try how that all works on Windows and how it interacts with vsync.

However, all this somehow feels wrong: I can't find it right now, but I think there are comments in the code stating that the tics are supposed to be incremented at 60Hz, and that this is supposed to be independent of the render framerate.
But OTOH, they obviously didn't implement (or at least ship) that in Doom3, so maybe they discarded that idea and just forgot to remove some stray comments.

On the other other hand, I don't see the point of that whole async thread for incrementing tics, no matter if the renderer is running in sync with the game or not.
If one wanted to use threads there, it would probably make more sense to run the gamecode and the renderer in different threads, than having an additional thread just for incrementing a counter every 16ms...

the console cursor also changes the flashing frequency with different values for com_gameHz

the console cursor also changes the flashing frequency with different values for com_gameHz

true, I've also noticed this already, will add it to the list (I also know why it happens, just doing other changes first)

I've pushed lots of changes in the last hours.
One thing is that I rebased this on the soft-particle branch, so both features can be tested together (I'll post Windows binaries later).

Another is that the framerates should now be very close to what's configured with com_gameHz (unless VSync or a slow computer slow it down), because the timing of frame starts is now done with much more precision than full milliseconds.

Most times in the game and engine still use full (integer) milliseconds, like before, so I wonder if this causes any issues or if starting the frames at the correct time is good enough for things to run smoothly.

looks goot so far, played up to 1st airlock in mars city underground.

There is an issue with the air supply though. It can be manipulated by changing com_gameHz while being outside

com_gameHz 500 -> cycle airlock then set com_gameHz to a lower value
com_gameHz 20, then again to 10 > 4000

changing com_gameHz to a higher value than configured has ofc the opposide effectl

(rebased this branch again, to current master, so it includes the fix for #587)

I'll look at the oxygen issue.
I don't think it's critical though, changing com_gameHz while outside is quite an edge-case ;)

Thanks for testing by the way, I appreciate that you're doing this, and how thorough you are! :)

By the way, does dhewm3 at higher framerates feel smooth?
(Or does your display only support 60Hz anyway so there's not much visual difference, unless a bug introduced additional stuttering?)

It was a coincidence, I changed the setting while i was still in the airlock, but after I already initiated the airlock cycle.
I wonder what other scripted/timed events could be prolonged in this manner?
I guess in the end you'll need a fps independent internal value/timer.
THe same happens with loading a savegame craeated with a different com_gameHz setting.
maybe keep a 60 FPS value around and save only this value in a savegame.

Desktop is set to 144Hz. I would not want to go back to 60Hz.
Higher FPS feel ok. No stutter, nothing feels very good so far. Tested on WIndows with 144Hz and tied 250 for a while as well. Ofc not much is happening at this stage of the game.
Unexpectedly, I had some time to spare, because the Austria - Turkey game was only on a strangly colored pay-TV channel. Sometimes it's an advantage to follow a game via audio stream only.

I guess in the end you'll need a fps independent internal value/timer.

I do, it's just that some code (incl. some scripts) uses the game tics that are incremented each frame, and at a higher framerate they increment faster..
The fixes so far have been to at least scale the amount of tics a function waits (or sets as a limit, for example for "then I'll be out of oxygen") by com_gameHz/60 (=> for 120fps wait for twice as many tics), but as you noticed if com_gameHz are modified after such a "target tic number" has been set, you get glitches.

But maybe I could change the logic to use a floating point tic number and then scale the added value each frame (at 120Hz add 0.5 instead of 1 tic), so the target tic would remain the same.. I just hope this isn't used by too much code besides the oxygen.

Or I'd have to re-evaluate going back to 60Hz tics for the game code and only render at a higher framerate, but I'm not sure if this is feasible (and if there is even a point to that: what use are 120fps if nothing changes every second frame? but maybe something can be done without increasing the tic, I'll have to look at how the prediction code for multiplayer works, it might be related to this..)

nothing feels very good so far

why not?

nothing feels very good so far

why not?

aaaaah. damn.

No stutter, nothing. Feels very good so far.

Here is a build for Windows: dhewm3-1.5.4pre-highfps_win32.zip
Note that it does not support any mods for now, only the base game and RoE.

This contains both the soft particles changes and this high FPS stuff.

Other new features:

• r_displayRefresh allows setting the display refresh rate for real fullscreen mode (unless you're using Wayland, it doesn't support that)
• com_showFPS can now be set to 2 to also show avg/min/max frame times in ms

Based on the comment from @dezo2 at #250 (comment) I tested d3xp Erebus4 and yes, that new Imp near the start of the level definitely doesn't spawn (or drop?) like it should when running at 120 or 144 or 240 fps (it still seems to work fine at 60).

It should come out of the black hole that arrow is pointing to:

It eventually is there, at least if I stand directly below that hole, but at 60fps it drops down that hole pretty soon after that panel from the ceiling falls down after entering the room.

Something else I noticed is that things completely fall apart when I change com_gameHz while the game is running (loading a savegame again fixes it).
No matter if I increase or decrease it, and if it's only by a little bit (like 120 -> 121), I get weird moving double images (note that despite the muzzle flash I'm not currently shooting):

No idea what's going on there, or if the issue is in the scripts or C++ code or whatever.
UPD: That double vision problem seems to be related to me messing with the slowmo state in idGameLocal::SetGameHz().
Still no idea about the Imp (only thing I noticed is that it most probably does spawn, as I heard it hissing).

I got finally some time before going back to work so I just want to thank you Daniel for working on this. I tested the base game briefly and the game movement seems fine. What I found so far using com_gameHz 200 and 240 with VSync on a 240Hz VRR monitor:

The crane at the start of map alphalabs3 doesn' t come up when I press the button and crashes the game (something about binding an object to itself). Which is strange because the chaingun firing rate is OK, so the scripts should be adjusted.

There is a slight problem with 240Hz using VSync in cutscenes, where the audio sync (lipsync) drifts after a while even with default com_fixedTic 0. This is minor and can be fixed by running the engine at 200Hz (5ms int) - you can test it with map mars_city2 at the start, the survivor up the ladder there speaks relativelly long.

I also noticed very brief frame skips when the Hz does not match the integer framerate like 240Hz and again they dissapear when using 200Hz or by setting com_fixedTic 1. The frequency of the skips seems to match the difference between int and float frametimes so at 240Hz 4.1666ms they are realtivelly far between.

The imp in RoE should come down if you kick the panel under the hole - seems like the spawn is somehow tied to the panel position as it lands differently when it falls down before the imp at FPS >60.

I will continue testing when I have some time.

The crane crash is not caused by the scripts after all, I tried it with my old code and it needs the SetSteerSpeed adjustment in Physics_AF.h, it was this line in the code:
void SetSteerSpeed(const float speed) { steerSpeed = speed * 60.0f / USERCMD_HZ; }

But I thought you did something similar it in your new code, so not sure what's going on.

Thanks for testing!

seems like the spawn is somehow tied to the panel position as it lands differently when it falls down before the imp at FPS >60

Hmm interesting theory - maybe the panel blocks the spawn position or something?

SetSteerSpeed(const float speed) { steerSpeed = speed * 60.0f / USERCMD_HZ; }

I just realized: I replaced that code with
SetSteerSpeed( const float speed ) { steerSpeed = speed * gameLocal.gameTicScale; }
but gameTicScale is gameHz/60 (the factor for "how many more tics do I need to wait for the same amount of time") - so it must be steerSpeed = speed / gameLocal.gameTicScale; instead.
Same for STOP_SPEED

Just pushed a commit that fixes the crane (I tested the crane and didn't have crashes with 120, 240 or 60fps)

Updated Windows build: dhewm3-1.5.4pre-highfps2_win32.zip
Most important fixes since last build: Crane works now, so does changing framerate in d3xp, cursor blinking speed is normal at all framerates, com_showFPS 2 shows avg/min/max frame times of all frames in last 0.5 seconds.

By the way, some suggestions for testing:

• Don't forget to increase the framerate ;)
  • In the new menu, which can usually be opened with F10, it's under Game Options -> Framerate
  • Enable com_showFPS to see if the game actually runs at that framerate (VSync might prevent it)
  • You don't necessarily need a display that supports >60Hz. You should not see much of a difference then (compared to how dhewm3 looked/ran before these changes), but most bugs should also be visible when running (without VSync) with 120 (or 144 or 240 or whatever) FPS on a 60Hz screen.
  • And of course testing if 60fps still behave like before is also valuable
• Save frequently (maybe even increase com_numQuicksaves) and if you run into a bug, make backups of the last one or two quicksaves, which can hopefully help to reproduce it
• Please report the map an issue happens in, ideally take a screenshot of the place, and run getviewpos or where in the console (and report the output) - it will print at what coordinate in the level you are.

Tested RoE yesterday with com_gameHz 200 and was able to run from start to finish without problems (apart from that one blocked imp in erebus4). No crashes or game breaking bugs, enemies went up/down the stairs to find me, grabber/slowmo/oxygen/enviro suit/machinery/elevators/bridges all worked as expected, so to me the code is already very usable. I will also try the base game but don't really expect any major problems there. BTW thanks for the F10 hidden menu tip, didn't even know about it.

As for the second blocked imp, it's in the base game, map recycling2 (-494.73 1410.72 -11.75) 190.2, right after interaction with the left screen. Imp should jump out of his closet, but with FPS >= 85 (11ms) he will be blocked with a panel he tried to kick out. So similar situation to than one in RoE. Not sure if this can be fixed without breaking something else in the physics code.
QuickSave.zip

I just wanna point out that the original Doom 3 can now run at higher fps than BFG edition. What a time to be alive in.

I may have found a fix for the blocked imps and potentially other enemies hidden behind wals and panels. This was caused by the STOP_SPEED in physics being too low for higher FPS, so by replacing a line in the source files Physics_RigidBody.cpp (base and d3xp):
const float STOP_SPEED = 10.0f / gameLocal.gameTicScale;
by this line:
const float STOP_SPEED = 10.0f * gameLocal.gameTicScale;
all the blockages seem to be fixed including the imp in RoE erebus4 and the one I posted yesterday. This should also fix the wild enemy ragdols and panels movements after explosions etc. Now at 200 FPS they all look similar to 60 FPS. Hopefully this won't break anything else. I will continue testing with this but I think this can be it.

Crap, celebrated too soon. It works with 200 FPS but not with 144 or 120. The physics code is seriously weird. The STOP_SPEED has to be the culprit for these problems.

I think I've found a unique bug in this branch, so I'll report here. Quick-saving during the PDA pickup notification will cause a persistent erroneous graphic as seen in the screenshot.

Hi Daniel; I was just curious where you're at with this - is there any chance we might get upcapped fps without physics problems or do you think this will be quite difficult if not impossible?

I rebased this branch to current master.
Shouldn't make much of a difference, except that maybe it runs a bit smoother due to changes done there (like not using the async thread to calculate frame times anymore) - and of course other non-framerate related features from the master branch are available here now, like better MD3 support.

Doing this wasn't trivial, because I merged and then changed some parts of this branch to master for the "smoother timing at 60Hz" feature, so it's possible that I broke something - testing is welcome! :)

Hi Daniel; I was just curious where you're at with this - is there any chance we might get upcapped fps without physics problems or do you think this will be quite difficult if not impossible?

I don't know.

Not sure why but this branch doesn't work with Sikkmod, game crashes after trying to load the opening FMV. 1.5.5 RC2 works okay.